remotemanager.utils.tokenizer module

Module to store the tokenizer class

class remotemanager.utils.tokenizer.Tokenizer(content: str)

generate_tokens() requires a file-like object to function

This class mimics this behaviour by raising StopIteration once the end of the content is reached.

tokenize() → list

Runs the tokenization

property tokens: list

Returns all stored tokens

list format is [(type, string), …]

property names: list

Returns the derived name list

property numbers: list

Returns derived number list

exchange_name(a: str, b: str)

Exchanges name a with name b

property source

Regenerate the source from the stored tokens

untokenize is… unreliable, since it focuses on the repeatability of the round trip

So we should reconstruct manually

Key tokens: 0: End marker 1: Name 2: Number 3: String 4: Newline 5: Indent 6: Dedent

Returns:

reconstructed source

Return type:

(str)